Responsibilities: • Become the gatekeeper of information; you will know where all the data goes, and who to call if the data is not there. • Set up tests and monitoring systems for pipeline failures. • Work with a variety of API’s and integrate their data into our warehouse. Work with our partners to make sure our system is up to date with their current versions. Think ETL, but way better! • Work with a NoSQL datastore, SQL Database (yes we use both) ,and learn how to marry the two together. You will use processes such as sharding and replication • Manage a messaging pipeline, and tap into our third party pipelines like SQS and Kafka. • Own the code you write, and deploy on AWS/Heroku. • Work with our Data Scientists and Engineers to create new datasets, experiments and technology
Skills & Qualifications: • You love technology, and think that things should be built rather than bought • You probably built some kind of app, and now want to make things scale to reach a larger audience. • Get frustrated by bad technology and want to change that. • Launched a box or two in AWS
Knowing all of these is awesome, some is good, but must know at least one. Think of this as a Venn diagram, you probably don't have all, but the more the better. If you are really strong in one of the following areas it would be great to chat:
General Coding / Developer: • Python/Ruby/Javascript/Bash/Shell • Node.js • Nginx
Data Engineering: • SQL (MySQL, PostgresSQL)/NoSQL datastores • Hadoop / Amazon Redshift • Queuing Systems/Kafka/SQS/Reddis
DevOps: • Datadog/ Cloud Monitoring tools • Docker/Jenkins/Dusty/Vagrant • Puppet/Chef