Week 3: Advanced Data Handling

Topics Covered:

Core Concepts and Implementation:

AI-Enhanced Data Analysis Setup:

Getting Started:

  1. Ollama Setup
  2. Optional Commercial APIs 
    # If you choose to purchase API access:
# 1. Create accounts at openai.com or anthropic.com
# 2. Purchase credits (student discounts may be available)
# 3. Create a .env file (never commit this!)
OPENAI_API_KEY=your_purchased_key
ANTHROPIC_API_KEY=your_purchased_key
  3. Install Required Packages 
    pip install python-dotenv pandasai pandas-ai dataprep autoviz

Using AI Tools:

Important Note: The code examples below are for demonstration purposes only. They illustrate the general approach but are not production-ready. You will need to:
  • Debug and adapt the code to your specific use case
  • Handle errors and edge cases
  • Test with your actual data structure
  • Refer to the latest documentation as APIs may change
# Example code - requires debugging and adaptation
# Using Ollama (Available to all students)
from pandasai import SmartDataframe
from pandasai.llm import Ollama

llm_local = Ollama(model="llama2")  # See Ollama docs for available models
df_local = SmartDataframe(your_dataframe, config={'llm': llm_local})  # Replace your_dataframe
result_local = df_local.chat('Generate summary statistics')

# If you've purchased API access:
from dotenv import load_dotenv
import os
load_dotenv()

# OpenAI example (if purchased)
from pandasai.llm import OpenAI
llm = OpenAI(api_token=os.getenv('OPENAI_API_KEY'))
df = SmartDataframe(your_dataframe, config={'llm': llm})  # Replace your_dataframe

Note: These examples assume certain package versions and configurations. Always check the current documentation and be prepared to debug integration issues.

Model Comparison:

Weekly Assignment

Due: End of Week 3

Tasks:

  1. Data Analysis Setup
    • Install pandas-ai and related packages
    • Configure Ollama access
    • Test basic functionality
  2. Financial Data Analysis
    • Load and clean sample financial data
    • Perform basic statistical analysis
    • Create time series visualizations
  3. AI-Enhanced Analysis
    • Use Ollama for data exploration
    • Generate automated insights
    • Compare with traditional analysis
Submit: As instructed in the weekly assignment

Week 3 Projects:

  1. Market Analysis (Using Ollama)
    • Build data processing pipeline
    • Implement natural language queries
    • Generate automated reports
    • Optional: Compare with commercial API results
  2. Data Processing Pipeline
    • Automate data cleaning with AI assistance
    • Create interactive analysis system
    • Implement quality checks
    • Generate comprehensive reports

Note: All course assignments can be completed using the provided Ollama setup-though not going to be perfect. Commercial APIs are optional but encourage exploration and at student's discretion.

Best Practices:

Additional Resources:

Check the department's GPU server status page for Ollama availability and usage guidelines. For those interested in commercial APIs, compare pricing and features before purchasing.

← Previous Week | Next Week →
← Back to Home