Data Lake Reliability Engineer

WHO WE ARE
Goldman Sachs is a leading global investment banking, securities and investment management firm that provides a wide range of services worldwide to a substantial and diversified client base that includes corporations, financial institutions, governments and high net-worth individuals.
Founded in 1869, it is one of the oldest and largest investment banking firms. The firm is headquartered in New York and maintains offices in London, Bangalore, Frankfurt, Tokyo, Hong Kong and other major financial centres around the world.
We are committed to growing our distinctive Culture and holding to our core values which always place our client's interests first. These values are reflected in our Business Principles, which emphasise integrity, commitment to excellence, innovation and teamwork.
TEAM OVERVIEW
The GS Data Lake is an industry leading cutting edge data lake implementation that is leveraged by a majority of GS business units. Our team of engineers build solutions to the most complex problems leveraging distributed technology stacks such as Hadoop, Apache Spark, Kafka, Elastic Search, Flink in combination with a flora of warehouse technologies such as Sybase IQ, IBM DB2, MemSQL, Snowflake, S3 and more to come.
Technology teams across the Firm are clients, participating in providing and consuming data to & from the lake. Developers on the team create and manage the software that manages the data in the lake, ensure entitlements are enforced appropriately, data is milestoned, and is available for query on multiple target warehouse platforms. This platform enables structuring, management, integration, control, discovery, usage, and governance of our Data Assets.
Data Lake Engineering is comprised of 3 teams and we are looking for the full range of engineers from leads to junior developer. Our recruiting process is aligned to find the right fit for you within these three areas of focus and we do want you to learn about each of them in order to align your interests with available roles. If you believe that you have a targeted interest, please do say so.
Data Lake Reliability Engineering team is a data driven engineering team whose focus is to engineer solutions around stability, scalability and system optimization for the lake. A key part of the role is to build infrastructure to measure and predict everything from client behaviours to future capacity needs. Core skills are: Python or Java, applied statistics and fundamental data science / ML principles / SRE principles
- Develop cutting edge technology stack to measure big data platform and enable a metric driven platform management strategy
- Continues metric driven work to improve, optimize and harden our data lake for continues growth and usage
- Transform reactive process to proactive process through engineering
- Apply data science practices to understand platform and user behaviours
- Apply machine learning practices to automate and optimize
Qualifications
- 7+ years of experience with Java or Python with a good grasp of development, Object Oriented Analysis and Design and testing best practices.
- Experience in leading small teams and manage careers
- Good understanding of distributed systems
- Working knowledge of scripting languages, Linux, Networking protocols, security and file systems
- Strong technical skills, analytical mindset, self-motivated, independent, creative, can solve interesting and sometimes difficult technical problems under time pressure and resource constraints
- Commercially focused; seeks to understand the requirements and how they will benefit our clients, stakeholders, and business
- Experience with all stages in the development lifecycle: inception, analysis, design, review, testing, and deployment
- Good sense of user interaction and usability design to provide an intuitive, seamless end user experience
- Experience building and sustaining long-term relationships with clients and colleagues in a diverse global organization
- Judgment to prioritize and escalate issues in order to influence objectives and outcomes.
- Excellent written and verbal communication skills, including experience working directly with both technical and non-technical stakeholders
- Experience from driving change through metrics
- Experience developing in Jupyter, Django, Flask, Angular
- Experience driving change though applied statistics, Machine Learning / Data Science
- Experience in fullstack n-dimensional incident management and root cause investigation
- Familiarity of SRE principles
- Preferred experience of one of the following: Hadoop, Spark, MapReduce, Flink, Elasticsearch, Sybase IQ, DB2, .