Educational requirements: Bachelor
English requirements: Competent English
Requirements for skilled employment experience for years: 1-3 years
Required residence status: Temporary visa, Permanent resident, Citizen
Accept remote work: unacceptable
Mandatory Skills:
At least 7 years of overall experience in Big Data Domain with extensive knowledge in Banking, Insurance, and Mortgage Domain. At least 3 years of experience in the Big Data technologies for Batch Implementation using Python, Spark, Hive, Scala, HQL, Hadoop, Phoenix, Hbase, Bash, PowerShell Must have strong experience with DDEP Analytics platform and metadata driven framework Apache ATLAS. Must have experience with code management tools such as Bitbucket, GitHub and SVN. At least 3 years of experience in NoSQL databases such as Hbase, Cassandra and Phoenix. Proficient in Azure cloud services – ADLS2, HD Insight, Azure SQL, Cosmos DB, Blob Storage, Databricks. Must have hands on devops deployment tools – Jenkins, JIRA, Maven, sbt, Azure Devops, docker. Must understand Azure Machine Learning Services and problem-solving skills using Reinforcement, Supervised and Unsupervised learning models.
Duties and Responsibilities:
As a Lead Big Data Engineer, you will be responsible for designing and implementing the Analytics platform solutions for large and complex projects. You will also be involved in improving the initiative within Cognizant Community that inspire significant improvement across Big Data practices. Work on ingesting, storing, processing, and analyzing large data sets which are useful for business analytics and reporting purposes. Create data governance policies and support data versioning processes to protect customer personal data and help business to adhere security standards. Investigate and analyze alternative solutions to data storing, processing etc. to ensure most streamlined approaches are implemented Translate complex technical and functional requirements into detailed design and document for solution endorsement to commence build and test. Mentoring the team members and guide them to develop business logics using software best practices. Build and refine a robust Machine Learning pipeline to ensure models are responsive to changes and trends in the data. Work closely with a data science team and implementing data analytics pipelines to help business driving the customer solutions to their problems. Work with Platform Architects on the design and deliver new products into both non-production and production clusters along with documentation. Monitor the production jobs and providing support to solve business problems. Create the automation code using automation utilities and help business saving cost by reducing the time efforts. Identify problems and discuss with business teams and propose solutions to solve the identified problems. Optimize Big Data pipelines using tuning performance techniques. Documenting the solutions and explaining the usage to both support team members as well as to the business team. Address and solve complex business problems using large amounts of data as inputs. Managing the Software Development Lifecycle using DevOps pipelines. Code defects fixes that are raised during QT and UAT Testing cycles. Build robust, efficient, and scalable data pipelines to ingest data from different sources into a cloud big data platform. Identify various data risks associated with the data flow process and document themitigation strategies. Cleanse the source raw data and transform and store into cloud cluster database for business analytical purpose.