RepoRankRepoRank
Join
Open navigation menu
Log inSign up

Pillar

Big Data Repositories & Open Source Data Infrastructure Projects

Explore the most popular big data repositories, large-scale processing tools, and open source data infrastructure projects. From distributed compute and storage systems to analytics engines, streaming platforms, and large-scale data workflows, discover which big data projects are gaining traction on GitHub.

Explore Big Data Topics

No active child topics are mapped to this pillar yet.

Trending Data Repositories

just now
  1. #1King of the HillDataPython

    RepoRank Score

    69

    scrapy/scrapy

    scrapyscrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    —61,317GitHub stars0boosts (24h)+14stars (24h)

Recent blogs

Stay Ahead

Get weekly Big Data repos in your inbox

Trending open-source projects, delivered weekly.

Get weekly Big Data repos in your inbox preview

Explore Open Source Big Data Projects

Big data tools are built to handle datasets, workflows, and infrastructure demands that go beyond traditional local or lightweight analytics environments. Open source repositories in this space help teams work with distributed processing, large-scale storage, stream-based systems, and the architecture needed for modern data-intensive applications.

The open source big data ecosystem includes distributed compute engines, streaming platforms, data storage systems, analytics frameworks, large-scale processing utilities, and broader repositories built for data-heavy infrastructure and operations. RepoRank helps surface the repositories that are earning real attention and momentum.

What You Will Find Here

  • Distributed compute and large-scale analytics repositories
  • Streaming systems, storage platforms, and data processing tools
  • Infrastructure projects for data-intensive engineering workflows
  • Emerging big data repositories gaining traction

This page helps you discover the big data tools engineers, platform teams, and analytics organizations are actively using, evaluating, and watching.

Why RepoRank Is Different

RepoRank focuses on real GitHub growth signals, helping you identify big data repositories that are active, relevant, and gaining adoption across large-scale data and analytics workflows.

  • Live GitHub star growth and activity tracking
  • A mix of established data infrastructure projects and rising repositories
  • A discovery layer built for practical large-scale data engineering

Built for Data Engineers, Platform Teams, and Analytics Organizations

Whether you are evaluating distributed processing tools, building scalable data infrastructure, or tracking open source repositories shaping modern big data workflows, this page helps you stay close to the projects driving large-scale data systems forward.

  • Data engineers working with large-scale processing and storage
  • Platform teams evaluating distributed data infrastructure
  • Organizations tracking fast-moving open source big data projects

Use this page to discover trending big data repositories, compare tools, and stay current with the open source projects shaping modern large-scale analytics and infrastructure.

Big Data FAQ

What are big data repositories?

Big data repositories are open source codebases related to large-scale data processing, storage, analytics, streaming, and distributed infrastructure.

What types of big data projects are included here?

This page includes distributed compute engines, analytics platforms, streaming systems, storage tools, large-scale processing frameworks, and broader open source repositories for data-intensive infrastructure.

How does RepoRank rank big data repositories?

RepoRank uses real GitHub growth signals such as star growth, activity, and project momentum to surface big data projects that are gaining traction.

Are these big data repositories open source?

Yes, all featured repositories are open source projects sourced directly from GitHub.

Why should I track trending big data repositories?

Tracking trending big data repositories helps you discover new infrastructure patterns, compare large-scale processing approaches, and evaluate the tools data teams are actively adopting.

What is the difference between big data and general data engineering tools?

Big data tools are typically focused on high-scale processing, distributed systems, and heavy storage or throughput demands, while general data engineering tools can also support smaller-scale pipelines, orchestration, and analytics workflows.

Are big data tools only useful for enterprise-scale teams?

No. While big data tools are often associated with scale-heavy environments, they are also useful for growing startups, modern data platforms, and teams preparing for more demanding workloads.

How do I choose the right big data repository?

Start with your scale, data patterns, and infrastructure needs. Consider performance model, operational complexity, ecosystem support, maintainability, documentation, and how well the repository fits your architecture.