GitHub - merico-dev/lake: DevLake: the data lake for dev & dashboard for you...
source link: https://github.com/merico-dev/lake
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
What is Dev Lake?
Dev Lake brings all your DevOps data into one practical, personalized, extensible view. Ingest, analyze, and visualize data from an ever-growing list of developer tools, with our free and open source product.
Dev Lake is most exciting for leaders and managers looking to make better sense of their development data, though it's useful for any developer looking to bring a more data-driven approach to their own practices. With Dev Lake you can ask your process any question, just connect and query.
Get started with just a few clicks
Dashboard Screenshot
User Flow
Why Dev Lake?
- Comprehensive understanding of software development lifecycle, digging workflow bottlenecks
- Timely review of team iteration performance, rapid feedback, agile adjustment
- Quickly build scenario-based data dashboards and drill down to analyze the root cause of problems
What can be accomplished with Dev Lake?
- Collect DevOps performance data for the whole process
- Share abstraction layer with similar tools to output standardized performance data
- Built-in 20+ performance metrics and drill-down analysis capability
- Support custom SQL analysis and drag and drop to build scenario-based data views
- Flexible architecture and plug-in design to support fast access to new data sources
See Demo
Click here to see demo. The demo is based on data from this repo.
Username/Password: test/test
Contents
Section Sub-section Description Documentation Link Data Sources Supported Data Sources Links to specific plugin usage & details View Section Setup Guide User Setup Set up Dev Lake locally as a user View Section Developer Setup Set up development environment locally View Section Cloud Setup Set up DevLake in the cloud with Tin View Section Tests Tests Commands for running tests View Section Make Contribution Understand the architecture of DevLake See the architecture diagram View Section Build a Plugin Details on how to make your own plugin View Section Add Plugin Metrics Guide to add metrics View Section Contribution specs How to contribute to this repo View Section User Guide, Help, and more Grafana How to visualize the data View Section Need Help Message us on Discord View Section FAQ Frequently asked questions by users View Section License The project license View SectionData Sources We Currently Support
Below is a list of data source plugins used to collect & enrich data from specific sources. Each has a README.md
file with basic setup, troubleshooting, and metrics info.
For more information on building a new data source plugin, see Build a Plugin.
Section Section Info Docs Jira Summary, Data & Metrics, Configuration, Plugin API Link GitLab Summary, Data & Metrics, Configuration, Plugin API Link Jenkins Summary, Data & Metrics, Configuration, Plugin API Link GitHub Summary, Data & Metrics, Configuration, Plugin API LinkSetup Guide
There're 3 ways to set up Dev Lake: user setup, developer setup and cloud setup.
User setup
- If you only plan to run the product locally, this is the ONLY section you should need.
- Commands written
like this
are to be run in your terminal.
Required Packages to Install
NOTE: After installing docker, you may need to run the docker application and restart your terminal
Commands to run in your terminal
-
Clone repository:
git clone https://github.com/merico-dev/lake.git devlake cd devlake cp .env.example .env
-
Start Docker on your machine, then run
docker-compose up -d
to start the services. -
Visit
localhost:4000
to setup configuration files.- Submit the form to update the values by clicking on the Save Connection button on each form page
devlake
takes a while to fully boot up. ifconfig-ui
complaining about api being unreachable, please wait a few seconds and try refreshing the page.- To collect this repo for a quick preview, please provide a Github personal token on Data Integrations / Github page.
-
Visit
localhost:4000/triggers
to trigger data collection.- Please refer to this wiki How to trigger data collection. Data collection can take a while depending on the amount of data you'd like to collect.
- To collect this repo for a quick preview, you can use the following JSON
[ [ { "Plugin": "github", "Options": { "repositoryName": "lake", "owner": "merico-dev" } } ], [ { "plugin": "github-domain", "options": {} } ] ]
-
Click View Dashboards button when done (username:
admin
, password:admin
). The button will be shown on the Trigger Collection page when data collection has finished.
Setup cron job
To synchronize data periodically, we provide lake-cli
for easily sending data collection requests along with a cron job to periodically trigger the cli tool.
Developer Setup
Requirements
- Docker
- Golang
- Make
- Mac (Already installed)
- Windows: Download
- Ubuntu:
sudo apt-get install build-essential
How to setup dev environment
-
Navigate to where you would like to install this project and clone the repository:
git clone https://github.com/merico-dev/lake.git cd lake
-
Install Go packages:
make install
-
Copy the sample config file to new local file:
cp .env.example .env
Find the line start with
DB_URL
in the file.env
and replacemysql:3306
with127.0.0.1:3306
-
Start the MySQL and Grafana containers:
Make sure the Docker daemon is running before this step.
docker-compose up mysql grafana
-
Run lake and config UI in dev mode in two seperate terminals:
# run lake make dev # run config UI make configure-dev
-
Visit config UI at
localhost:4000
to configure data sources.- Submit the form to update the values by clicking on the Save Connection button on each form page
-
Visit
localhost:4000/triggers
to trigger data collection.- Please refer to this wiki How to trigger data collection. Data collection can take up to 20 minutes for large projects. (GitLab 10k+ commits or Jira 5k+ issues)
-
Click View Dashboards button when done (username:
admin
, password:admin
). The button is shown in the top left.
Cloud setup
If you want to run Dev Lake in a clound environment, you can set up Dev Lake with Tin. See detailed setup guide
Disclaimer:
To protect your information, it is critical for users of the Tin hosting to set passwords to protect Dev Lake applications. We built Dev Lake as a self-hosted product, in part to ensure users have total protection and ownership of their data, while the same remains true for the Tin hosting, this risk point can only be eliminated by the end-user.
Tests
To run the tests:
make test
Make Contribution
This section list all the documents to help you contribute to the repo.
Understand the Architecture of DevLake
Architecture Diagram
Add a Plugin
Add Plugin Metrics
Contributing Spec
User Guide, Help and more
Grafana
We use Grafana as a visualization tool to build charts for the data stored in our database. Using SQL queries, we can add panels to build, save, and edit customized dashboards.
All the details on provisioning and customizing a dashboard can be found in the Grafana Doc.
Need help?
Message us on Discord
Q: When I run docker-compose up -d
I get this error: "qemu: uncaught target signal 11 (Segmentation fault) - core dumped". How do I fix this?
A: M1 Mac users need to download a specific version of docker on their machine. You can find it here.
License
This project is licensed under Apache License 2.0 - see the LICENSE
file for details.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK