IBM Data Engineer
IBM
- Cairo
- Permanent
- Full-time
IBM Systems Expert Labs is a Professional Services Team under IBM Technology Expert Labs.
IBM Technology Expert Labs has the proven expertise to help leaders plan, design and implement the essential IT infrastructure for what comes next. We offer a wide range of IT infrastructure services for IBM Power Systems™, IBM Z® and IBM LinuxONE™ systems, and IBM Storage & Software Defined Infrastructure. Technology Expert Labs – Systems’ consultants perform IT infrastructure services for clients on site, helping them solve business challenges, gain new skills and discover best practices. Expert Lab has a global presence and can deploy its consultants around the world.
- As a Storage Consultant, the role involves providing specialised storage services around the world. The Expert Labs specialist is a highly skilled expert with a deep knowledge of IBM's Storage Software portfolio and hardware products
- The Expert Labs consultant assume overall technical responsibility for the success of solution construction, implementation and system integration in a technology, industry or business specialty
- You deliver new and complex high-quality solutions to clients in response to varying business requirements
- You have deep skill and strong hands-on skills in ONE or MORE of the following subject matter areas: Enterprise Storage, IBM DS8000, IBM Flash System Products, SAN Networking and Tape products, Spectrum Protect, Spectrum Scale
- Exposure to Containers and Kubernetes technology
- Should be ready to travel (once or twice a month)
As an IBM data engineer, you will be responsible for designing, constructing, installing, testing, and maintaining highly scalable data management systems. You will work closely with data architects, data scientists, and business analysts to understand data requirements, design data models, and implement solutions that optimize data ingestion, storage, and processing.
Key Responsibilities:
1. Data Pipeline Development: Develop and maintain data pipelines to ingest, transform, and load structured and unstructured data from various sources into data storage systems such as data lakes, data warehouses, and databases.
2. Data Modeling: Design and implement efficient data models to support analytical and operational needs, ensuring data integrity, accuracy, and consistency across different data sets.
3. Data Integration: Integrate data from multiple sources, including internal databases, external APIs, third-party data providers, and streaming data sources, using appropriate tools and technologies.
4. Data Processing: Implement data processing workflows using tools like Apache Spark, Hadoop, or IBM Cloud Pak for Data to perform ETL (Extract, Transform, Load) operations, data cleansing, and data enrichment.
5. Database Management: Manage and optimize databases, including relational databases (e.g., IBM Db2, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra), to ensure high performance, availability, and scalability.
6. Data Quality Assurance: Develop and implement data quality checks, validation rules, and monitoring processes to ensure data accuracy, completeness, and consistency over time.
7. Performance Tuning: Optimize data pipelines, queries, and processes for performance, scalability, and efficiency, leveraging techniques such as indexing, partitioning, and caching.
8. Collaboration: Collaborate with cross-functional teams, including data scientists, business analysts, software engineers, and stakeholders, to understand data requirements, prioritize tasks, and deliver data-driven solutions.
9. Documentation: Create and maintain documentation for data pipelines, data models, data dictionaries, and technical specifications to ensure transparency, repeatability, and knowledge sharing within the team.Required Technical and Professional Expertise
1. Bachelor's or Master's degree in computer science, information technology, data engineering, or a related field.
2. Proficiency in programming languages such as Python, Java, Scala, or SQL for data manipulation, scripting, and automation.
3. Experience with data pipeline orchestration tools like Apache Airflow, IBM DataStage, or Informatica.
4. Strong knowledge of database management systems (DBMS), data warehousing concepts, and data modeling techniques.
5. Familiarity with big data technologies such as Apache Hadoop, Apache Spark, Kafka, and distributed computing frameworks.
6. Experience with cloud platforms like IBM Cloud, AWS, Azure, or Google Cloud Platform for data storage, processing, and analytics.
7. Excellent problem-solving skills, analytical thinking, and attention to detail in designing and troubleshooting data solutions.
8. Effective communication skills to collaborate with diverse teams, present technical findings, and explain complex concepts to non-technical stakeholders.
9. Ability to work independently, prioritize tasks, and adapt to changing project requirements in a dynamic environment.Preferred Technical and Professional Expertise
1. Certification in data engineering or related technologies (e.g., IBM Certified Data Engineer, AWS Certified Data Analytics - Specialty).
2. Experience with data visualization tools like Tableau, Power BI, or IBM Cognos for creating interactive dashboards and reports.
3. Knowledge of machine learning concepts, data science workflows, and statistical analysis techniques.
4. Familiarity with agile development methodologies, DevOps practices, and version control systems (e.g., Git).
5. Experience with containerization technologies such as Docker and orchestration tools like Kubernetes for deploying and managing data applications.