Horje
Read large SAS file ilarger than memory n Python Code Example
Read large SAS file ilarger than memory n Python 
import pandas as pd
import pyreadstat
filename = 'foo.SAS7BDAT'
CHUNKSIZE = 50000
offset = 0
allChunk,_ = getChunk(row['filePath'], row_limit=CHUNKSIZE, row_offset=offset)
allChunk = allChunk.astype('category')

while True:
    offset += CHUNKSIZE
    # for xpt data, use pyreadstat.read_xpt()
    chunk, _ = pyreadstat.read_sas7bdat(filename, row_limit=CHUNKSIZE, row_offset=offset)
    if chunk.empty: break  # if chunk is empty, it means the entire data has been read, so break

    for eachCol in chunk:  #converting each column to categorical 
        colUnion = pd.api.types.union_categoricals([allChunk[eachCol], chunk[eachCol]])
        allChunk[eachCol] = pd.Categorical(allChunk[eachCol], categories=colUnion.categories)
        chunk[eachCol] = pd.Categorical(chunk[eachCol], categories=colUnion.categories)

    allChunk = pd.concat([allChunk, chunk])  #Append each chunk to the resulting dataframe
Source: stackoverflow.com



Python

Related
os Code Example os Code Example
get random bright hex color python Code Example get random bright hex color python Code Example
generate key password Code Example generate key password Code Example
Update only keys in python Code Example Update only keys in python Code Example
network setting for virtualbox kali Code Example network setting for virtualbox kali Code Example

Type:
 Code Example
Category:
 Coding
Sub Category:
 Code Example
Uploaded by:
 Admin
Views:
 9



 

Horje