• Yue Chen's avatar
    Fix sse overflow in highbd_*_obmc_variance sse4 version · e4e586d7
    Yue Chen authored
    Also the final sse will be stored as an unsigned 64-bit integer,
    which is enough. The cumulation process is done in 4 x int32.
    Therefore, for 10-bit and 12-bit video, int32 is not enough to
    hold 1/4 of the total sse of large blocks.
    In this patch, max_pel_allowed_per_ovf is set up for 10-bit and
    12-bit cases, hbd_obmc_variance_w8n is applied to patches sizes at
                       bw * (max_allowed_per_ovf / bw)
    to avoid overflow.
    
    BUG=aomedia:1772
    
    Change-Id: Iaae2e1403baa93c4ba454d6760281f95428f437a
    e4e586d7
Name
Last commit
Last update
..
aom_asm_stubs.c Loading commit data...
aom_convolve_copy_sse2.asm Loading commit data...
aom_high_subpixel_8t_sse2.asm Loading commit data...
aom_high_subpixel_bilinear_sse2.asm Loading commit data...
aom_subpixel_8t_intrin_avx2.c Loading commit data...
aom_subpixel_8t_intrin_ssse3.c Loading commit data...
aom_subpixel_8t_sse2.asm Loading commit data...
aom_subpixel_8t_ssse3.asm Loading commit data...
aom_subpixel_bilinear_sse2.asm Loading commit data...
aom_subpixel_bilinear_ssse3.asm Loading commit data...
avg_intrin_sse2.c Loading commit data...
avg_ssse3_x86_64.asm Loading commit data...
blend_a64_hmask_sse4.c Loading commit data...
blend_a64_mask_sse4.c Loading commit data...
blend_a64_vmask_sse4.c Loading commit data...
blend_sse4.h Loading commit data...
common_avx2.h Loading commit data...
convolve.h Loading commit data...
convolve_avx2.h Loading commit data...
convolve_common_intrin.h Loading commit data...
convolve_sse2.h Loading commit data...
convolve_sse4_1.h Loading commit data...
fwd_dct32_8cols_sse2.c Loading commit data...
fwd_dct32x32_impl_avx2.h Loading commit data...
fwd_dct32x32_impl_sse2.h Loading commit data...
fwd_txfm_avx2.c Loading commit data...
fwd_txfm_avx2.h Loading commit data...
fwd_txfm_impl_sse2.h Loading commit data...
fwd_txfm_sse2.c Loading commit data...
fwd_txfm_sse2.h Loading commit data...
fwd_txfm_ssse3_x86_64.asm Loading commit data...
halfpix_variance_impl_sse2.asm Loading commit data...
halfpix_variance_sse2.c Loading commit data...
highbd_convolve_avx2.c Loading commit data...
highbd_convolve_ssse3.c Loading commit data...
highbd_intrapred_sse2.asm Loading commit data...
highbd_intrapred_sse2.c Loading commit data...
highbd_loopfilter_avx2.c Loading commit data...
highbd_loopfilter_sse2.c Loading commit data...
highbd_quantize_intrin_avx2.c Loading commit data...
highbd_quantize_intrin_sse2.c Loading commit data...
highbd_sad4d_sse2.asm Loading commit data...
highbd_sad_sse2.asm Loading commit data...
highbd_subpel_variance_impl_sse2.asm Loading commit data...
highbd_subtract_sse2.c Loading commit data...
highbd_variance_impl_sse2.asm Loading commit data...
highbd_variance_sse2.c Loading commit data...
highbd_variance_sse4.c Loading commit data...
intrapred_avx2.c Loading commit data...
intrapred_sse2.asm Loading commit data...
intrapred_sse2.c Loading commit data...
intrapred_ssse3.c Loading commit data...
inv_txfm_avx2.c Loading commit data...
inv_txfm_common_avx2.h Loading commit data...
inv_txfm_sse2.c Loading commit data...
inv_txfm_sse2.h Loading commit data...
inv_txfm_ssse3.c Loading commit data...
inv_wht_sse2.asm Loading commit data...
jnt_sad_ssse3.c Loading commit data...
jnt_variance_ssse3.c Loading commit data...
loopfilter_avx2.c Loading commit data...
loopfilter_sse2.c Loading commit data...
lpf_common_sse2.h Loading commit data...
masked_sad_intrin_ssse3.c Loading commit data...
masked_variance_intrin_ssse3.c Loading commit data...
masked_variance_intrin_ssse3.h Loading commit data...
mem_sse2.h Loading commit data...
obmc_intrinsic_ssse3.h Loading commit data...
obmc_sad_sse4.c Loading commit data...
obmc_variance_sse4.c Loading commit data...
quantize_avx_x86_64.asm Loading commit data...
quantize_sse2.c Loading commit data...
quantize_ssse3_x86_64.asm Loading commit data...
sad4d_avx2.c Loading commit data...
sad4d_sse2.asm Loading commit data...
sad_avx2.c Loading commit data...
sad_highbd_avx2.c Loading commit data...
sad_impl_avx2.c Loading commit data...
sad_sse2.asm Loading commit data...
ssim_opt_x86_64.asm Loading commit data...
subpel_variance_sse2.asm Loading commit data...
subtract_sse2.asm Loading commit data...
sum_squares_sse2.c Loading commit data...
synonyms.h Loading commit data...
synonyms_avx2.h Loading commit data...
transpose_sse2.h Loading commit data...
txfm_common_avx2.h Loading commit data...
txfm_common_intrin.h Loading commit data...
txfm_common_sse2.h Loading commit data...
variance_avx2.c Loading commit data...
variance_impl_avx2.c Loading commit data...
variance_sse2.c Loading commit data...