Bi9 dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZmZdd lmZe e Zejejed Zd ed efdZd e fdZ!de fdZ"dede#de#ded ef dZ$dede#de#fdZ%de&ed ej'fdZ(ded e)e&ee&ee ffdZ*ded efdZ+dZ,e-d kr e,dSdS)!zz Smart PDF ingestion system with incremental processing. Only processes new or modified files, preserving existing index. N)tqdm)OpenAI)Path)datetime)OPENAI_API_KEY EMBED_MODEL CHUNK_SIZE CHUNK_OVERLAP INDEX_DIR TENANT_ID)load_pdf_pagesextract_section_heading) FaissStore)api_keyzprocessed_files.jsonfilepathreturncVtj} t|d5tfddD]}|| dddn #1swxYwY|S#t $r }td|d|Yd}~dSd}~wwxYw)z Calculate MD5 hash of file to detect changes. Args: filepath: Path to file Returns: MD5 hash as hex string rbc.dS)Ni )read)fsHC:\Users\Terasoftware\OneDrive\Desktop\Graph Rag\rag_suite\rag\ingest.pyzget_file_hash../saffTllrNu ⚠️ Warning: Could not hash : )hashlibmd5openiterupdate hexdigest Exceptionprint)rhash_md5chunkers @r get_file_hashr("s{}}H (D ! ! 'Q2222C88 ' '&&&& ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '!!###  @@@Q@@AAArrrrrs:A>,A A>A##A>&A#'A>> B(B##B(c.tjtrp t tdd5}t j|cdddS#1swxYwYn+#t$r}td|icYd}~Sd}~wwxYwiS)z Load record of already processed files from disk. Returns: Dict mapping filename -> {hash, processed_date, chunks, pages} rutf-8encodingNu/⚠️ Warning: Could not load tracking file: ) ospathexists TRACKING_FILErjsonloadr#r$)rr's rload_processed_filesr47s w~~m$$ mS7;;; $qy|| $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $    GAGG H H HIIIIII  Is@A*A A*A""A*%A"&A** B4B B B processedc\ tjtjtdt tdd5}t j||ddd d d d S#1swxYwYd S#t$r}td |Yd }~d Sd }~wwxYw) zu Save record of processed files to disk. Args: processed: Dict mapping filename -> metadata T)exist_okwr+r,Findent ensure_asciiNu/⚠️ Warning: Could not save tracking file: ) r.makedirsr/dirnamer1rr2dumpr#r$)r5rr's rsave_processed_filesr@HsE BGOOM22TBBBB -w 7 7 7 B1 Ii15 A A A A B B B B B B B B B B B B B B B B B B EEE CCCDDDDDDDDDEs<ABA7* B7A;;B>A;?B B+B&&B+doc_idpage chunk_ordertextc |d|d|d|dd}tj|dS)a Generate stable, deterministic chunk ID. Args: doc_id: Document identifier page: Page number chunk_order: Chunk sequence number text: Chunk text (first 50 chars used) Returns: SHA1 hash as hex string |N2r+)rsha1encoder")rArBrCrDbases rstable_chunk_idrK[sW 7 7t 7 7k 7 7D"I 7 7D < G,, - - 7 7 9 99r chunk_sizeoverlapc#PKd|}|sdSd}|t|krctt|||z}|||V|t|krdS||z }|dkr|}|t|kadSdS)a Generate text chunks with overlap using generator (memory efficient). Args: text: Input text to chunk chunk_size: Size of each chunk in characters overlap: Overlap between consecutive chunks Yields: Text chunks as strings  Nr)joinsplitlenmin)rDrLrMstartends rchunk_text_generatorrVls 88DJJLL ! !D  E #d))  #d))UZ/0059o #d))   Eg  19 E #d))      rtextsc|stjddS tjt |}d|jD}tj|dS#t$r}td|d}~wwxYw)z Generate embeddings for text chunks using OpenAI API. Args: texts: List of text strings to embed Returns: Numpy array of shape (len(texts), embedding_dim) )rifloat32)dtype)modelinputcg|] }|j S) embedding).0ds r zembed_texts..s999aak999ru!❌ Error generating embeddings: N) npzerosclient embeddingscreaterdataarrayr#r$)rWresponserfr's r embed_textsrks 4x 3333 $++,  :98=999 x )4444  5!55666 sA A'' B 1BB foldercttjtj|d}|st d|t }g}g}|D]}tj|}t|}|s| |H||vr`|| dd}||kr| |td|| |td|| ||||fS)z Analyze which files need processing. Args: folder: Path to folder containing PDFs Returns: Tuple of (files_to_process, skipped_files, processed_files_dict) *.pdfzNo PDF files found in hashru 🔄 Modified: u ✨ New: ) sortedglobr.r/rP ValueErrorr4basenamer(appendgetr$) rlpdfsprocessed_filesfiles_to_process skipped_filespdf_path file_name current_hash stored_hashs ranalyze_files_to_processr~sn $)BGLL99:: ; ;D <:&::;;;+,,OM..G$$X.. $X..    # #H - - -   ' .))488DDK{* 2$$Y////6966777 ''1111 ,,, - - -  # #H - - - - ]O ;;rc v td|dt|\}}}t|t|z}tdtd|tdt|tdt||rdtd|dd D]}td |t|d kr#td t|d z d |s-td tdttStdt|dt jtrt|dkrtd tj t}tdt|j dn#t$rV}td|tdtdgj d}t|}Yd}~nBd}~wwxYwtdtdgj d}t|}g} d} t|dD]} t j| } t!j| } t)| }|std| d}|D]}|d}|d}|s't-|}t/|t0t2D]f}|st5| |||}t|g}t6| | | |||||d }|||g|dz }g| |z } t;| t=j |t|d!|| <| !| | t||d"#t$r=}td#t j| d$|Yd}~d}~wwxYwtd%|"ttd&ttG|td'tt j$t j%td(} tM|d)d*+5}tOj(| |d,d-.dddn #1swxYwYtd/|n)#t$r}td0|Yd}~nd}~wwxYwtS)1z Ingest PDFs from folder with incremental processing. Only processes new or modified files. Args: folder: Path to folder containing PDFs Returns: Path to saved index directory u 📂 Scanning folder:  u📊 Ingestion Plan:z - Total PDFs found: z$ - Already processed (unchanged): z - New or modified: u" ⏭️ Skipping unchanged files:Nu ✓ z ... and z moreu1 ✅ All files already up-to-date! Nothing to do.u& 💡 To reprocess all files, delete: u 🔄 Processing z file(s)... ru📂 Loading existing index...u" ✅ Loaded existing index with z chunks u* ⚠️ Could not load existing index: z! Creating new index instead... zdimension probe)dimu📂 Creating new index... zProcessing PDFs)descu* ⚠️ Warning: No pages extracted from rBrD) tenant_idrAdoc_name source_url page_numbersection_headingchunk_idrCrD)roprocessed_datechunkspages)rArrru ❌ Error processing ru 💾 Saving updated index...u ✅ Index saved to: u ✅ Tracking file updated: zingestion_manifest.jsonr8r+r,r9Fr:u ✅ Manifest saved: u$ ⚠️ Could not save manifest: ))r$r~rRr1r r.r/r0rr3metar#rkshaperrsrrrIr"r striprrVr r rKr addr(rnow isoformatrtsaver@rPr>rr2r?)rlrxryrw total_pdfsnamestorer'ringestion_manifesttotal_chunks_addedrzrrAr chunk_counterrBpage_numrDrr&rvecr manifest_pathrs ringest_folder_incrementalrs /V / / /0008PPV7W7W4m_%&&]););;J !""" 0J 0 0111 E]1C1C E EFFF :3'7#8#8 : :;;;B 4555!"1"% ' 'D %t%% & & & & }   ! B @3}#5#5#9@@@ A A A  CDDD G GGHHH Cs#344 C C CDDD w~~i  $S%7%7!%; $ ./// (OI..E Qs5:QQQ R R R R ( ( ( BqBB C C C 6 7 7 701228;C3'''EEEEEE  ( ,---,-..4Q7s###)0ABBBGGF w''11H[!2!233==??F#8,,E NHNNOOOM" '" '<F|zz||"9$"?"?2$ MRR''E ;;==! .vxPUVVH&ug..C&/"($,&.'/+:$,'4 %  DIIcD6***!Q&MM1'4 - / &h//"*,..":":"<"<'U ))OH %  % % $U' ''        MBG,<,G I*A H;;IA:QD0Q R  2RR 6V U-! V -U11V 4U15V V1V,,V1ctdtdtdd}tj|s3td|dtdtdd St jtj|d }|s3td |d td tdd S t |}tdtdtdtj|rQ tj|}tdtdt|j dn#YnxYwt}tdt|tdtd|tdttdtdtdtd|dtdd S#t$r}tdtdtdtd|ddd l}td |tdYd }~d Sd }~wwxYw)!z&Main entry point for ingestion script.z= ============================================================u%🚀 RAG INCREMENTAL INGESTION SYSTEMz<============================================================rhu ❌ Error: Folder 'z ' not found!z& Please create it and add PDF files.z============================================================= Nrnu ❌ No PDF files found in 'z'!z& Please add PDF files to the folder.u✅ INGESTION COMPLETE!u 📊 Final Index Statistics:z - Total chunks in index: ,z - Total tracked files: u 📁 Output Locations:z - Index: z - Tracking: u 💡 Next Steps:z 1. Test Q&A: python -m rag.qaz) 2. Start API: uvicorn api:app --reloadz 3. Add more PDFs to 'z/' and run againz> ============================================================ u❌ INGESTION FAILED!z Error: rrzFull traceback:)r$r.r/r0rqrPrrr3rRrr4r1r# traceback print_exc)rl pdf_files index_pathrr5r'rs rmainrs / 1222 (OOO F 7>>& ! ! :f:::;;; 7888 o "',,vw7788I  8f888999 7888 o*&.v66  o '((( h 7>>* % %  " 337888HS__HHHIIII )**  ;3y>>;;<<< )*** )Z))*** / //000 #$$$ 1222 :;;; AAAABBB $%%%%%  & & & o %&&& h !        $%%%%%%%%% &s4*AI AFI FB9I KA6KK__main__).__doc__r.rqrr2numpyrcropenairpathlibrr rag.configrrr r r r rag.pdf_utilsr rrag.vectorstorerrer/rPr>r1strr(dictr4r@intrKrVlistndarrayrktupler~rr__name__r^rrrs   ccccccccccccccccAAAAAAAA&&&&&&  ' ' ' RW__Y779OPP CC*d" ED E E E E&:C:s::C:C::::" s  c    FtCyRZ:0