20 most asked #interview #question in #spark with #answers

1. What is Spark?
   Spark is an open-source distributed computing system used for processing large-scale data sets. It provides high-level APIs for programming in Java, Scala, Python, and R.

2. What are the key features of Spark?
   The key features of Spark include in-memory processing, support for a wide range of data sources, and built-in support for machine learning, graph processing, and streaming data processing.

3. What is an RDD in Spark?
   RDD (Resilient Distributed Datasets) is the fundamental data structure in Spark. It is an immutable distributed collection of objects, which can be processed in parallel across multiple nodes.

4. What are the different transformations in Spark?
   The different transformations in Spark include map, filter, flatMap, distinct, groupByKey, reduceByKey, sortByKey, join, and union.

5. What are the different actions in Spark?
   The different actions in Spark include collect, count, first, take, reduce, save, foreach, and foreachPartition.

6. What is lazy evaluation in Spark?
   Lazy evaluation is a feature in Spark where the transformations are not executed until an action is called. This reduces unnecessary computations and improves performance.

7. What is the difference between map and flatMap in Spark?
   Map applies a function to each element in a RDD and returns a new RDD, while flatMap applies a function that returns an iterator to each element in a RDD and returns a flattened RDD.

8. What is the difference between transformation and action in Spark?
   A transformation is a function that produces a new RDD from an existing one, while an action is a function that returns a result or saves data to a storage system.

9. What is Spark Streaming?
   Spark Streaming is a component of Spark that allows processing of real-time data streams using Spark's batch processing engine.

10. What is Spark SQL?
    Spark SQL is a module in Spark that allows processing of structured and semi-structured data using SQL-like queries.

11. What is Spark MLlib?
    Spark MLlib is a machine learning library in Spark that provides scalable implementations of various machine learning algorithms.

12. What is a broadcast variable in Spark?
    A broadcast variable is a read-only variable that can be cached on each machine in a cluster for more efficient data sharing.

13. What is SparkContext in Spark?
    SparkContext is the entry point for Spark applications and represents the connection to a Spark cluster.

14. What is the role of the Driver program in Spark?
    The Driver program is the main program that defines the transformations and actions to be performed on the data.

15. What is a cluster manager in Spark?
    A cluster manager is responsible for managing the resources and scheduling the tasks across the nodes in a Spark cluster.

16. What is a Shuffle in Spark?
    A Shuffle is the process of redistributing data across the nodes in a cluster to prepare it for a subsequent operation, such as a reduce operation.

17. What is a Partition in Spark?
    A Partition is a logical unit of data in a RDD that can be processed in parallel across different nodes.

18. What is a DAG in Spark?
    A DAG (Directed Acyclic Graph) is a data structure in Spark that represents the sequence of transformations and actions to be executed on a RDD.

19. What is a Spark Executor?
    A Spark Executor is a worker node in a Spark cluster that executes tasks on behalf of the Driver program.

20. What is a Spark Worker?
    A Spark Worker is a node in a Spark cluster that runs Executors and manages the resources allocated to them.