o Ci;@sddlZddlTddlZddZddZedfddZd&d d Zd'd d Z e Z d&ddZ ddZ e Zd(ddZ ddZddZeZd)ddZeZd)ddZGdddZd*d d!Zefd"d#ZGd$d%d%ZdS)+N)*cCtj|dd}|j\}}tj||fdd}tj||fdd}t}t||_t||_||_ ||_ | | |t|| ||fS)zPreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDharl/var/www/html/fyndo/python/python_agents/rag_suite/venv/lib/python3.10/site-packages/faiss/extra_wrappers.pykmin   rcCr)zOreturn k largest values (and their indices) of the lines of a float32 arrayrrr)rr r r r float_minheap_array_trrrrrrrrrrrrkmax(r r"c Cstj|dd}tj|dd}|j\}}|j\}}||ksJtj||fdd}|tkr compute a checksum for quick-and-dirty comparisons of arrays uint8Nint32iy)viewr7 ivec_checksumintrrangex)arn4csirrrchecksumks $rPcCs(tj||fdd}t||t|||Sr5)rr#rand_smooth_vectors_cr)rr.r9r:rrrrand_smooth_vectorsysrSc Cstj|dd}tj|dd}|jd}|jd|ksJ|jd|jd}}d}t|D]}|t|t|||t||7}q-|S)z< size of intersection between each line of two result tablesrrr)rr r rJranklist_intersection_sizer)I1I2rk1k2ninterrOrrreval_intersections  r[cCs t|jd|jdt|dS)NrTr)fvec_renorm_L2r r)rKrrr normalize_L2s r]cCs|tj|dd}|durt|d}tj|ddd}tj|jdd}t|jt| d|t|t||||fS)aPerform a bucket sort on a table of integers. Parameters ---------- tab : array_like elements to sort, max value nbucket - 1 nbucket : integer number of buckets, None if unknown nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) perm : array_like perm[lims[i] : lims[i + 1]] contains the indices of bucket #i (size tab.size) rrNrTuint64) rr rImaxr#r7 bucket_sort_cr rrG)tabnbucketntlimspermrrr bucket_sortsrfcCsn|jdks |jdks J|j\}}|durt|d}tj|ddd}t||t||t|||S)aPerform a bucket sort on a matrix, recording the original row of each element. Parameters ---------- tab : array_like array of size (N, ncol) that contains the bucket ids, maximum value nbucket - 1. On output, it the elements are shuffled such that the flat array tab.ravel()[lims[i] : lims[i + 1]] contains the row numbers of each bucket entry. nbucket : integer number of buckets (the maximum value in tab should be nbucket - 1) nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) rFrNrTr) rr rIr_rr#matrix_bucket_sort_inplace_cr r)rarbrcnrowncolrdrrrmatrix_bucket_sort_inplaces   rjc@s2eZdZdZd ddZddZddZd d Zd S) ResultHeapz_Accumulate query results from a sliced dataset. The final result will be in self.D, self.I.FcCs~tj||fdd|_tj||fdd|_|||_|_|r!t}nt}||_||_t |j|_ t |j|_ | ||_ dS)z nq: number of query vectors, k: number of results per query keep_max: keep the top-k maximum values instead of the minima rrrN)rr rrr-rr!r rrrrrheaps)selfr-rkeep_maxrlrrr__init__s   zResultHeap.__init__cCsd|j\}}tj|dd}tj|dd}|j||fksJ||jks#J|j|t|t||dS)z Add results for all heaps D, I should be of size (nh, nres) D, I do not need to be in a particular order (heap or sorted) rrrN)r rr r-rl addn_with_idsr)rmrrr-kdrrr add_results zResultHeap.add_resultcCs|j\}}|t|ks J|jdkr|j|jks%|jdkr#|j|fks%Jtj|dd}tj|dd}tj|dd}|jdkrAdn|}|j|t||t|t||dS)z Add results for a subset of heaps. D, I should hold resutls for all the subset as a special case, if I is 1D, then all ids are assumed to be the same rTrrrrN)r lenndimrr rladdn_query_subset_with_idsr)rmsubsetrrnsubsetrq id_striderrradd_result_subsets zResultHeap.add_result_subsetcCs|jdS)N)rlr)rmrrrfinalizeszResultHeap.finalizeNF)__name__ __module__ __qualname____doc__rorrrzr{rrrrrks   rkFc Cs||j|jksJ|j\}}}tj||f|jd}tj||f|jd}|r&tnt}||||t|t|t|t|||fS)z Merge a set of sorted knn-results obtained from different shards in a dataset Dall and Iall are of size (nshard, nq, k) each D[i, j] should be sorted returns D, I of size (nq, k) as the merged result set r)r rr#rmerge_knn_results_CMaxmerge_knn_results_CMinr) DallIallrnnshardrrDnewInewfuncrrrmerge_knn_resultss    rc Cstj|dd}tj|dd}|j\}}|j\}}||ksJtj||fdd}tj||fdd} |tkrKtt|t|||||t| t|| |fS|tkrftt|t|||||t| t|| |fSt d)a Compute the k nearest neighbors of a vector without constructing an index Parameters ---------- xq : array_like Query vectors, shape (nq, d) where d is appropriate for the index. `dtype` must be float32. xb : array_like Database vectors, shape (nb, d) where d is appropriate for the index. `dtype` must be float32. k : int Number of nearest neighbors. distance_type : MetricType, optional distance measure to use (either METRIC_L2 or METRIC_INNER_PRODUCT) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rrrz'only L2 and INNER_PRODUCT are supported) rr r r#r$ knn_L2sqrrr&knn_inner_productNotImplementedError) r)r*rr+r-r.r/r0rrrrrknn0s(      rc@s*eZdZdZddZd ddZddZdS) KmeansaObject that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator cKs||_||_d|_d|vrt|_nt|_|D]$\}}|dkr1|dks*|dkr-t}||_qt|j|t |j||qd|_ dS)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False Fprogressive_dim_stepsgpuTN) r.rr"ProgressiveDimClusteringParameterscpClusteringParametersitems get_num_gpusgetattrsetattr centroids)rmr.rkwargsvrrrros   zKmeans.__init__Nc stj|dd}|j\}}||jksJ|jjtkr^t||j|j}|dur9|j\}}||ks0Jt | |j |jj rCt||_nt||_|jrUt j|j|jd|_|||j|n0|dusdJ|dusjJ|jj rpJt||j|j}|jrt|jd} nt} ||t|| t |j } | |j||_ |jfddtDtddD|_dfd dD|_|jjd kr|jd Sd S) a  Perform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rrN)ngpucsg|]}|qSr)at).0rO)statsrr sz Kmeans.train..cSsg|]}|jqSr)obj)rstrrrrsz,obj time time_search imbalance_factor nsplitcsg|] fddDqS)csi|]}|t|qSr)r)rfieldrrr sz+Kmeans.train...r)r) stat_fieldsrrrsrrg) rr r r.r __class__r Clusteringrr copy_array_to_vectorravelr spherical IndexFlatIPindex IndexFlatL2rindex_cpu_to_all_gpustrainProgressiveDimClusteringGpuProgressiveDimIndexFactoryProgressiveDimIndexFactoryrvector_float_to_arrayreshapeiteration_statsrJr7rrsplit) rmrKweightsinit_centroidsrr.clusncr0facrr)rrrrsB           z Kmeans.traincCsZtj|dd}|jdusJd|j|j|j|j|d\}}||fS)Nrrzshould train before assigningrT)rr rrresetaddsearchr)rmrKrrrrrassigns  z Kmeans.assign)NN)r}r~rrrorrrrrrrfs   Fr)r3)r3N)rQ)Nrr|)numpyr faiss.loaderr rr"r$r2r;r@lrandrBrPrSrRr[r]rfr`rjrgrkrrrrrrrs.       ' @ 6