Home
TidyDataCLI is a powerful command-line tool designed to streamline the process of cleaning, transforming, visualizing, and reporting on Excel and CSV data.
It is particularly useful for data analysts, researchers, and anyone working with tabular data who needs an efficient way to prepare data for analysis.
The tool adheres to the principles of "tidy data," ensuring datasets are structured for easy analysis.
TidyDataCLI is cross-platform, running seamlessly on Linux, macOS, Windows, and via Docker, making it accessible without requiring a local Python installation.
TidyDataCLI offers a comprehensive set of features categorized into four main areas:
Data Cleaning, Data Transformation, Visualization, and Report Generation.
Runs on Linux, macOS, Windows, and Docker, ensuring flexibility across different environments.
To follow the project and it's releases visit github.
Installation
Python 3.7 or higher: Required for native installation.Pip: Python package manager for installing dependencies.Docker (Optional): For containerized execution.
The simplest way to install TidyDataCLI is using pip.
pip install TidyDataCLI
To install from the source code:
Clone the repository: git clone https://github.com/Siam3h/TidyDataCLI
Navigate to the repository directory: cd tidydatacli
Install the package: pip install
For users preferring a containerized environment:
Pull the Docker image: docker pull tidydatacli
Run the tool, mounting the current directory to,
Example: /data:docker run -v $(pwd):/data tidydatacli tidydata <command> --input /data/input.csv --output /data/output.csv
Usage
TidyDataCLI supports four primary commands:
clean: Performs data cleaning tasks like removing duplicates or standardizing formats.
transform: Applies transformations such as sorting, filtering, or adding columns.
visualize: Generates visual representations like charts or word clouds.
report: Creates reports in text or PDF format.
For a complete list of options.
tidydata <command> --help
Remove Duplicates: Eliminates duplicate rows to ensure data integrity.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Regex Cleaning: Uses regular expressions to remove or replace unwanted patterns in text data.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Column Name Cleaning: Standardizes column names by removing spaces, special characters, or converting to a consistent case.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Trim Spaces: Removes leading and trailing spaces from text fields.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Age Validation: Validates age data to ensure it falls within a specified range.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Change Case: Converts text to upper, lower, or title case for consistency.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Date Standardization: Converts dates to a uniform format (e.g., YYYY-MM-DD).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Sorting: Sorts data by one or more columns in ascending or descending order.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Filtering: Extracts subsets of data based on user-defined conditions.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Custom Transformations: Applies custom lambda functions for advanced data manipulation.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Column Addition: Creates new columns based on calculations or existing data.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Aggregation: Performs summary operations like sum, mean, count, min, or max.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
TidyDataCLI supports various visualization types to help users explore and communicate data insights:
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Bar Charts: Compare categorical data (e.g., sales by region).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Pie Charts: Show proportions (e.g., market share).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Word Clouds: Visualize text data by highlighting frequent terms.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Line Charts: Display trends over time (e.g., monthly revenue).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Box-and-Whisker Plots: Show data distribution and outliers.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Gantt Charts: Visualize project timelines or schedules.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Heat Maps: Highlight patterns in numerical data (e.g., correlation matrices).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Histograms: Display the distribution of numerical data (e.g., age distributions).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Tree Maps: Represent hierarchical data (e.g., organizational structures).
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Generates reports in text or PDF format, customizable with summary statistics, visualizations, or detailed data tables.
tidydata clean standardize-date input.csv --column 'Join Date' --output 'standardized_dates.csv'
Tutorials
This section provides step-by-step guides for common tasks using TidyDataCLI.
Assuming a sample dataset data.csv with columns name, age, date, category, and value.
To remove duplicate rows and standardize column names
(e.g., converting "Customer Name" to "customer_name"):
tidydata clean --input data.csv --output cleaned_data.csv --remove_duplicates --clean_columns
Trim SpacesTo remove leading/trailing spaces from text fields:
tidydata clean --input data.csv --output cleaned_data.csv --remove_duplicates --clean_columns --trim_spaces
Standardize DatesTo convert dates to a uniform format (e.g., YYYY-MM-DD):
tidydata clean --input data.csv --output cleaned_data.csv --remove_duplicates --clean_columns --trim_spaces --standardize_dates
Validate Age DataTo ensure age values are within a reasonable range (e.g., 0–120):
tidydata clean --input data.csv --output cleaned_data.csv --validate_age
Best Practices
To maximize the effectiveness of TidyDataCLI:
Always work on a copy of your original dataset to prevent data loss.
Name output files clearly (e.g., cleaned_data_2025-06-21.csv) to track processing steps.
Regularly visit the GitHub repository for new features or bug fixes.
For team workflows or cross-system use, leverage Docker to avoid dependency issues.
Use cleaning and validation options (e.g., --validate_age) to catch errors before transformations.
Ensure columns selected for visualizations match the chart type (e.g., numerical data for histograms).
Record commands and transformations for reproducibility, especially in complex projects.
For large files, monitor system resources and consider splitting-objcopy
Troubleshooting
Verify the file path and ensure the file exists.
Confirm the file is a valid CSV or Excel file and not corrupted.
Check syntax using tidydata <command> --help.
For large datasets, use Docker or ensure sufficient system memory.
Ensure selected columns exist and match the expected data type (e.g., numerical for histograms)
For unresolved issues, visit the Github Issues Page or contact the. maintainer, siama.
TidyDataCLI provides informative error messages for common issues, such as file not found, invalid formats, or incorrect command usage.
Always consult the tidydata <command> --help. option for correct syntax and refer to the troubleshooting section for guidance.
Contributing
To contribute:
Fork the repository on Github.
Create a branch for your changes.
Submit a pull request with your improvements or bug fixes.
Report issues or suggest features via the Github Issues Page.
Refer to the Github repository’s contributing guidelines for detailed instructions.
TidyDataCLI is released under the MIT License.
For support or feedback, contact the maintainer at Philbert Siama.
Developed by Philbert Siama, TidyDataCLI aims to provide a user-friendly, powerful tool for data preparation. It is actively maintained and open to community contributions.