Thesis work in machine learning

Job description
Master’s thesis in machine learning: Geometrical invariances in computationally efficient transformer neural network architectures


Thesis type: 30 credits
Project start: January 2024
Student location: On Site, Gothenburg Sweden



Abstract


Transformer architectures (attention mechanisms) is a rather new way of approaching dependencies in space/time. Prior to attention one needed to propagate information through many layers of a neural network and make rather restrictive assumptions on the spatial dependency structure to model long-term dependencies. With transformers it is possible to find a direct and flexible connection (ideally) within just one layer. This has dramatically enhanced the performance of models in many fields, such as computer vision and natural language processing, and has laid the foundation for much of the breakthroughs in AI/ML during the last years (like chat-GPT for instance). 


Although transformers have many positive properties, like most things they also have some drawbacks. One of these is that the "attention" mechanism requires evaluating a matrix between all "tokens" at the same time. This means that it scales quadratically in both memory and processing. For computer vision problems, this drawback makes it infeasible to work with large transformer-based networks without having access to heavy computational resources. This problem is further amplified if one wants to perform analysis in space and time (moving images).


Due to this restriction, strong AI-models are so far only available to the few with strong computational resources and a large energy budget, making it a concern for equity and environment as well as a bottleneck for scientific and engineering progress. In this thesis the student will do research, perform analysis, develop code, and make (and evaluate) algorithms for sparse representations of the attention mechanism. Emphasis should be put on incorporating different types of geometric invariances into such a sparse representation.


At AstraZeneca, all of our employees make a difference to patients’ lives every day. We operate in more than 100 countries around the world and are one of Sweden’s most important export companies.


With more than 2,400 employees from 50 countries, the vibrant Gothenburg site helps to support the entire life-cycle of AstraZeneca medicines, from drug discovery and clinical trials, to global commercialisation and product maintenance. Gothenburg is one of AstraZeneca’s three strategic, global R&D centres, alongside Cambridge and Gaithersburg, and plays a central role in our mission to deliver life-changing medicines to patients.


Application
Randstad Life Sciences is collaborating with AstraZeneca in this recruitment process. Please send your application in English and we only accept applications through Randstad’s website.


Deadline for application: 2023-11-03, selection and interviews will be ongoing. The position may be filled before the last day of application, therefore, apply as soon as possible.


All chosen candidates need to pass a background check and an alcohol and drug test before starting the project. 

Responsibilities
In this thesis the student will do research, perform analysis, develop code, and make (and evaluate) algorithms for sparse representations of the attention mechanism. Emphasis should be put on incorporating different types of geometric invariances into such a sparse representation. 

Qualifications
The student need to have experience of programming in a machine learning neural network framework (like PyToch, TensorFlow or similar). The student should also have a strong general programming background as well as a good understanding of linear algebra, statistics, and mathematical modelling. We see that the student is pursuing a degree in mathematics, physics, computer science, data science, or some similar quantitative field. 

About the company
If you’re inspired by the possibilities of science to make a difference and ready to discover what you can do – join us! 


At AstraZeneca we are guided in our work by a strong set of values, and we’re resetting expectations of what a bio-pharmaceutical company can be. By truly following the science, we pioneer new methods, new thinking and bring unexpected teams together. From scientists to sales, lab techs to legal, we’re on a mission to turn ideas into life-changing medicines that transform lives. We need great people who share our passion for science and have the drive and determination to meet the unmet needs of patients around the world. If you’re swift to action, confident to lead, willing to collaborate, and curious about what science can do, then you’re our kind of person.


AstraZeneca is an equal opportunity employer. AstraZeneca will consider all qualified applicants for employment without discrimination on grounds of disability, sex or sexual orientation, pregnancy or maternity leave status, race or national or ethnic origin, age, religion or belief, gender identity or re-assignment, marital or civil partnership status, protected veteran status (if applicable) or any other characteristic protected by law. AstraZeneca only employs individuals with the right to work in the country/ies where the role is advertised.