The algorithm prints out a incorrect p-value for 'aov'.
Sample code:
library(Devore7)
motor_stack<-stack(ex10.37)
motor_aov<-aov(values~ind, motor_stack)
summary(motor_aov)
correct p-value is '0.000187' whereas one calculated by R in solus is '0.0855'
Are you sure this is a Solus issue and not an issue with R itself or the CRAN package used? If you test this on another machine not running Solus do you receive the correct p-value?
I only ask because I don't see anyway that Solus itself could be causing this issue.
I ran the same set of code on Windows 10 and MacOs. The same was run on "https://rdrr.io/snippets/". All three outputs are same and correct. I don't know the internal build and architecture of solus to pinpoint this issue but it seems the issue is with the rstudio built for solus or solus itself.
I am getting the correct value.
> library(Devore7)
Loading required package: MASS
Loading required package: lattice
> motor_stack<-stack(ex10.37)
> motor_aov<-aov(values~ind, motor_stack)
> summary(motor_aov)
Df Sum Sq Mean Sq F value Pr(>F)
ind 4 30.86 7.714 8.444 0.000187 ***
Residuals 25 22.84 0.914
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1@gs , can you post the full output you recieve if you open R in terminal and run the codes? Since you arrive at the correct value perhaps it is an issue with a particular version of R?
Below is my output:
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-solus-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library(Devore7)
Loading required package: MASS
Loading required package: lattice
> motor_stack<-stack(ex10.37)
> motor_aov<-aov(values~ind, motor_stack)
> summary(motor_aov)
Df Sum Sq Mean Sq F value Pr(>F)
ind 4 1696 423.9 2.312 0.0855 .
Residuals 25 4583 183.3
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Today moving further, linear model is also not learning from the given data.
Check for ex12. 15 in Devore7, MoE dependence on Strength.
I don't think that R is build properly for solus.
@joebonrichie you did the last update to R 3.6.1, is there anything I could give you (console outputs, etc.) that might help in chasing down this issue, as your time permits of course.
@adurante, here you have the full output
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-solus-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library(Devore7)
Loading required package: MASS
Loading required package: lattice
> motor_stack<-stack(ex10.37)
> motor_aov<-aov(values~ind, motor_stack)
> summary(motor_aov)
Df Sum Sq Mean Sq F value Pr(>F)
ind 4 30.86 7.714 8.444 0.000187 ***
Residuals 25 22.84 0.914
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1@gs Awesome thank you. That confirms were on the same version.
I ran the R install tests and it seems that stats is installed incorrectly.
tools::testInstalledPackages(scope = "base")
Below is my output which indicates the tests are failing on the stats package (which makes sense since that's where the AOV function is located)
R version 3.6.1 (2019-07-05) -- "Action of the Toes" Copyright (C) 2019 The R Foundation for Statistical Computing Platform: x86_64-solus-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > tools::testInstalledPackages(scope = "base") Testing examples for package ‘base’ Testing examples for package ‘tools’ Testing examples for package ‘utils’ Running vignettes for package ‘utils’ Testing examples for package ‘grDevices’ Testing examples for package ‘graphics’ Testing examples for package ‘stats’ Error: testing 'stats' failed >
I will try uninstalling and reinstalling R and Rstudio later today to confirm if that is a possible fix to the issue since there is no way to reinstall an R-base package
@adurante Even I received same error output upon running the test. There are issues with linear model as well. Kindly look into the same as well.
Reinstall did not fix the issue. Same error persists and test on "stats" package still fails
@gs sadly I thought of that and I tested R on my machine after reinstalling but before reinstalling rstudio. Was a good shout though.
I am now messing about with update.packages() and getting dependency errors that I am hoping rectifying may solve this issue.
@gs could be completely unrelated but I just thought of this. Reinstalling R does not purge other CRAN packages. It is possible one of them may have corrupted the stats package.
Is your R install a clean install? Or have you installed additional packages with install.packages()?
Alright, well, I purged my entire R install of all packages, then removed R and Rstudio via the software center, and then deleted their directories as sudo.
Then I reinstalled, confirmed it was a clean base installation, however the issue remains....
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-solus-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> installed.packages()
Package LibPath Version Priority
base "base" "/usr/lib64/R/library" "3.6.1" "base"
boot "boot" "/usr/lib64/R/library" "1.3-22" "recommended"
class "class" "/usr/lib64/R/library" "7.3-15" "recommended"
cluster "cluster" "/usr/lib64/R/library" "2.1.0" "recommended"
codetools "codetools" "/usr/lib64/R/library" "0.2-16" "recommended"
compiler "compiler" "/usr/lib64/R/library" "3.6.1" "base"
datasets "datasets" "/usr/lib64/R/library" "3.6.1" "base"
foreign "foreign" "/usr/lib64/R/library" "0.8-71" "recommended"
graphics "graphics" "/usr/lib64/R/library" "3.6.1" "base"
grDevices "grDevices" "/usr/lib64/R/library" "3.6.1" "base"
grid "grid" "/usr/lib64/R/library" "3.6.1" "base"
KernSmooth "KernSmooth" "/usr/lib64/R/library" "2.23-15" "recommended"
lattice "lattice" "/usr/lib64/R/library" "0.20-38" "recommended"
MASS "MASS" "/usr/lib64/R/library" "7.3-51.4" "recommended"
Matrix "Matrix" "/usr/lib64/R/library" "1.2-17" "recommended"
methods "methods" "/usr/lib64/R/library" "3.6.1" "base"
mgcv "mgcv" "/usr/lib64/R/library" "1.8-28" "recommended"
nlme "nlme" "/usr/lib64/R/library" "3.1-140" "recommended"
nnet "nnet" "/usr/lib64/R/library" "7.3-12" "recommended"
parallel "parallel" "/usr/lib64/R/library" "3.6.1" "base"
rpart "rpart" "/usr/lib64/R/library" "4.1-15" "recommended"
spatial "spatial" "/usr/lib64/R/library" "7.3-11" "recommended"
splines "splines" "/usr/lib64/R/library" "3.6.1" "base"
stats "stats" "/usr/lib64/R/library" "3.6.1" "base"
stats4 "stats4" "/usr/lib64/R/library" "3.6.1" "base"
survival "survival" "/usr/lib64/R/library" "2.44-1.1" "recommended"
tcltk "tcltk" "/usr/lib64/R/library" "3.6.1" "base"
tools "tools" "/usr/lib64/R/library" "3.6.1" "base"
utils "utils" "/usr/lib64/R/library" "3.6.1" "base"
Depends
base NA
boot "R (>= 3.0.0), graphics, stats"
class "R (>= 3.0.0), stats, utils"
cluster "R (>= 3.3.0)"
codetools "R (>= 2.1)"
compiler NA
datasets NA
foreign "R (>= 3.0.0)"
graphics NA
grDevices NA
grid NA
KernSmooth "R (>= 2.5.0), stats"
lattice "R (>= 3.0.0)"
MASS "R (>= 3.1.0), grDevices, graphics, stats, utils"
Matrix "R (>= 3.2.0)"
methods NA
mgcv "R (>= 2.14.0), nlme (>= 3.1-64)"
nlme "R (>= 3.4.0)"
nnet "R (>= 2.14.0), stats, utils"
parallel NA
rpart "R (>= 2.15.0), graphics, stats, grDevices"
spatial "R (>= 3.0.0), graphics, stats, utils"
splines NA
stats NA
stats4 NA
survival "R (>= 2.13.0)"
tcltk NA
tools NA
utils NA
Imports LinkingTo
base NA NA
boot NA NA
class "MASS" NA
cluster "graphics, grDevices, stats, utils" NA
codetools NA NA
compiler NA NA
datasets NA NA
foreign "methods, utils, stats" NA
graphics "grDevices" NA
grDevices NA NA
grid "grDevices, utils" NA
KernSmooth NA NA
lattice "grid, grDevices, graphics, stats, utils" NA
MASS "methods" NA
Matrix "methods, graphics, grid, stats, utils, lattice" NA
methods "utils, stats" NA
mgcv "methods, stats, graphics, Matrix, splines, utils" NA
nlme "graphics, stats, utils, lattice" NA
nnet NA NA
parallel "tools, compiler" NA
rpart NA NA
spatial NA NA
splines "graphics, stats" NA
stats "utils, grDevices, graphics" NA
stats4 "graphics, methods, stats" NA
survival "graphics, Matrix, methods, splines, stats, utils" NA
tcltk "utils" NA
tools NA NA
utils NA NA
Suggests
base "methods"
boot "MASS, survival"
class NA
cluster "MASS, Matrix"
codetools NA
compiler NA
datasets NA
foreign NA
graphics NA
grDevices "KernSmooth"
grid "lattice"
KernSmooth "MASS"
lattice "KernSmooth, MASS, latticeExtra"
MASS "lattice, nlme, nnet, survival"
Matrix "expm, MASS"
methods "codetools"
mgcv "parallel, survival, MASS"
nlme "Hmisc, MASS"
nnet "MASS"
parallel "methods"
rpart "survival"
spatial "MASS"
splines "Matrix, methods"
stats "MASS, Matrix, SuppDists, methods, stats4"
stats4 NA
survival NA
tcltk NA
tools "codetools, methods, xml2, curl, commonmark"
utils "methods, xml2, commonmark"
Enhances License
base NA "Part of R 3.6.1"
boot NA "Unlimited"
class NA "GPL-2 | GPL-3"
cluster NA "GPL (>= 2)"
codetools NA "GPL"
compiler NA "Part of R 3.6.1"
datasets NA "Part of R 3.6.1"
foreign NA "GPL (>= 2)"
graphics NA "Part of R 3.6.1"
grDevices NA "Part of R 3.6.1"
grid NA "Part of R 3.6.1"
KernSmooth NA "Unlimited"
lattice "chron" "GPL (>= 2)"
MASS NA "GPL-2 | GPL-3"
Matrix "MatrixModels, graph, SparseM, sfsmisc" "GPL (>= 2) | file LICENCE"
methods NA "Part of R 3.6.1"
mgcv NA "GPL (>= 2)"
nlme NA "GPL (>= 2) | file LICENCE"
nnet NA "GPL-2 | GPL-3"
parallel "snow, nws, Rmpi" "Part of R 3.6.1"
rpart NA "GPL-2 | GPL-3"
spatial NA "GPL-2 | GPL-3"
splines NA "Part of R 3.6.1"
stats NA "Part of R 3.6.1"
stats4 NA "Part of R 3.6.1"
survival NA "LGPL (>= 2)"
tcltk NA "Part of R 3.6.1"
tools NA "Part of R 3.6.1"
utils NA "Part of R 3.6.1"
License_is_FOSS License_restricts_use OS_type MD5sum
base NA NA NA NA
boot NA NA NA NA
class NA NA NA NA
cluster NA NA NA NA
codetools NA NA NA NA
compiler NA NA NA NA
datasets NA NA NA NA
foreign NA NA NA NA
graphics NA NA NA NA
grDevices NA NA NA NA
grid NA NA NA NA
KernSmooth NA NA NA NA
lattice NA NA NA NA
MASS NA NA NA NA
Matrix NA NA NA NA
methods NA NA NA NA
mgcv NA NA NA NA
nlme NA NA NA NA
nnet NA NA NA NA
parallel NA NA NA NA
rpart NA NA NA NA
spatial NA NA NA NA
splines NA NA NA NA
stats NA NA NA NA
stats4 NA NA NA NA
survival NA NA NA NA
tcltk NA NA NA NA
tools NA NA NA NA
utils NA NA NA NA
NeedsCompilation Built
base NA "3.6.1"
boot "no" "3.6.1"
class "yes" "3.6.1"
cluster "yes" "3.6.1"
codetools "no" "3.6.1"
compiler NA "3.6.1"
datasets NA "3.6.1"
foreign "yes" "3.6.1"
graphics "yes" "3.6.1"
grDevices "yes" "3.6.1"
grid "yes" "3.6.1"
KernSmooth "yes" "3.6.1"
lattice "yes" "3.6.1"
MASS "yes" "3.6.1"
Matrix "yes" "3.6.1"
methods "yes" "3.6.1"
mgcv "yes" "3.6.1"
nlme "yes" "3.6.1"
nnet "yes" "3.6.1"
parallel "yes" "3.6.1"
rpart "yes" "3.6.1"
spatial "yes" "3.6.1"
splines "yes" "3.6.1"
stats "yes" "3.6.1"
stats4 NA "3.6.1"
survival "yes" "3.6.1"
tcltk "yes" "3.6.1"
tools "yes" "3.6.1"
utils "yes" "3.6.1"
> tools::testInstalledPackages(scope = "base")
Testing examples for package ‘base’
Testing examples for package ‘tools’
Testing examples for package ‘utils’
Running vignettes for package ‘utils’
Testing examples for package ‘grDevices’
Testing examples for package ‘graphics’
Testing examples for package ‘stats’
Error: testing 'stats' failed
>As soon as time permits I will try installing a clean install of solus on a VM and installing R there. However at this time I am completely stumped.
Soooo. On a completely brand new solus OS 4.0 budgie within a VM with R installed all the tests succeed.
I also reinstalled all the *-devel packages I have on my current install in order to try and replicate my setup in the VM and the tests still succeed. So I am now out of ideas on how to pinpoint what is causing the corruption of the base "stats" package.
If anyone has any ideas on what else I can do to test/pinpoint/fix (or in the case of the VM, break) the issue let me know and i will be happy to try it out
This issue seems to be at multiple points in R build for linux. Here are my observations:
Note: all modules are working fine for windows and macos.
Right, but at the same time @adurante was able to clear the bug with a full reinstall in a VM. That says to me that this is either hardware-specific or user-specific.
I've been keeping an eye on this and haven't been able to reproduce any of the incorrect calculations thus far. We'll need to figure out the common denominator between the people having issues.
This is a long shot, but if your machine supports AVX2 maybe the avx2 libs of glibc's math libraries are producing wrong results (check with lscpu | grep avx2). I think it's much more likely to be an upstream issue with R itself if @auspicrypt can produce incorrect results with Ubuntu's R as well.
I may have misunderstood but I ran your above command and here is the output:
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
Let me know if there is another command I can run. I hope to have more time in the upcoming week to try and break my VM install haha. If you have any other things you would like me to check since my R seems to be broken in the same way I hope if I can fix the issue it fixes it for everyone
@adurante can you try removing glibc's avx2 libraries and then running a calculation again which produces wrong results. Do NOT run this on a working machine, keep it to a VM.
Bear in mind this is a long shot, as this is the only variable I can think of currently which might be different between people having issues or not.
@joebonrichie I have been trying to break my VM install of R so that I can attempt this fix on my VM rather than my work machine but I have been unsuccessful. Do you know if there is a way to copy the entirety of my work machine's R install to the VM (in theory bringing the bug with it) so that I can attempt this fix there?
Could the problem be related to this? https://bugzilla.kernel.org/show_bug.cgi?id=205663
@adurante I'm trying to get R to run correctly on my VM but it still won't work. Can you give me pointers?
I installed the budgie images on a VM ran only "required updates", then installed R (not Rstudio). A simple lm() simulate give incorrect results. Is that what you did or did you update other packages?
If this VM is not a work machine (I. E. If you kill the VM and have to reinstall it wouldn't be a big deal) could you try running @joebonrichie 's commands from above regarding glibc?
I have still failed to arrive at a broken state within my VM and don't yet want to start running such commands against my work machine
@joebonrichie @adurante I loaded a VM with solus-budge image from the website. Updated "required packages", installed R & dependencies, confirmed lm() gives incorrect results, ran @joebonrichie 's commands from above on glibc, ran lm() again --> still incorrect results.
Not sure what's wrong with the compilation wondering if @gs 's https://bugzilla.kernel.org/show_bug.cgi?id=205663 bug has something to do with it at this point... no idea...
@kccit and @gs and @auspicrypt
Can you run inxi -Fz so that @joebonrichie can compare and see if maybe there are hardware differences? I will do similar once I get to my machine tomorrow.
I will also try reinstalling the solus budgie image on my VM to see if I can arrive at the error that way
Here is the output from inxi -Fz on the VM *after* running the rm... glibc command. R gave bad results before and after the command.
System:
Host: solus_test Kernel: 4.20.16-112.current x86_64 bits: 64 Desktop: Budgie 10.5 Distro: Solus 4.0
Machine:
Type: Kvm System: QEMU product: Standard PC (i440FX + PIIX, 1996) v: pc-i440fx-4.1 serial: <filter> Mobo: N/A model: N/A serial: N/A BIOS: SeaBIOS v: rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org date: 04/01/2014
CPU:
Topology: Single Core model: Intel Core (Skylake IBRS) bits: 64 type: MCP L2 cache: 16.0 MiB Speed: 2904 MHz min/max: N/A Core speed (MHz): 1: 2904
Graphics:
Device-1: Red Hat QXL paravirtual graphic card driver: qxl v: kernel Display: x11 server: X.Org 1.20.3 driver: none unloaded: fbdev,modesetting,vesa resolution: 1024x768~60Hz OpenGL: renderer: llvmpipe (LLVM 7.0 256 bits) v: 3.3 Mesa 19.0.0
Audio:
Device-1: Intel 82801FB/FBM/FR/FW/FRW High Definition Audio driver: snd_hda_intel Sound Server: ALSA v: k4.20.16-112.current
Network:
Device-1: Intel 82371AB/EB/MB PIIX4 ACPI type: network bridge driver: piix4_smbus Device-2: Intel 82540EM Gigabit Ethernet driver: e1000 IF: ens3 state: up speed: 1000 Mbps duplex: full mac: <filter>
Drives:
Local Storage: total: 20.00 GiB used: 5.28 GiB (26.4%) ID-1: /dev/sda vendor: QEMU model: HARDDISK size: 20.00 GiB
Partition:
ID-1: / size: 18.64 GiB used: 5.28 GiB (28.3%) fs: ext4 dev: /dev/sda2 ID-2: swap-1 size: 953.7 MiB used: 0 KiB (0.0%) fs: swap dev: /dev/sda1
Sensors:
Message: No sensors data was found. Is sensors configured?
Info:
Processes: 139 Uptime: 4m Memory: 3.85 GiB used: 477.4 MiB (12.1%) Shell: bash inxi: 3.0.32
Alright, for me:
System with broken R (Not VM):
System:
Host: adurante-p7xxtm1 Kernel: 5.3.10-134.current x86_64 bits: 64 Desktop: Budgie 10.5.1 Distro: Solus 4.0
Machine:
Type: Laptop System: Notebook product: P7xxTM1 v: N/A serial: <filter> Mobo: Notebook model: P7xxTM1 serial: <filter> UEFI: American Megatrends v: 1.05.05RLS2 date: 11/23/2017
Battery:
ID-1: BAT0 charge: 76.4 Wh condition: 76.4/82.9 Wh (92%)
CPU:
Topology: 6-Core model: Intel Core i7-8700K bits: 64 type: MT MCP L2 cache: 12.0 MiB Speed: 900 MHz min/max: 800/4700 MHz Core speeds (MHz): 1: 900 2: 900 3: 900 4: 900 5: 900 6: 900 7: 900 8: 900 9: 900 10: 900 11: 900 12: 900
Graphics: Device-1:
NVIDIA GP106BM [GeForce GTX 1060 Mobile 6GB] driver: nvidia v: 440.31 Display: x11 server: X.Org 1.20.5 driver: nvidia tty: N/A OpenGL: renderer: GeForce GTX 1060/PCIe/SSE2 v: 4.6.0 NVIDIA 440.31
Audio:
Device-1: Intel 200 Series PCH HD Audio driver: snd_hda_intel Device-2: NVIDIA GP106 High Definition Audio driver: snd_hda_intel Sound Server: ALSA v: k5.3.10-134.current
Network:
Device-1: Qualcomm Atheros Killer E2500 Gigabit Ethernet driver: alx IF: enp109s0 state: up speed: 1000 Mbps duplex: full mac: <filter> Device-2: Intel Wireless 8265 / 8275 driver: iwlwifi IF: wlp112s0 state: up mac: <filter>
Drives:
Local Storage: total: 1.36 TiB used: 166.72 GiB (11.9%) ID-1: /dev/sda vendor: Seagate model: ST1000LX015-1U7172 size: 931.51 GiB ID-2: /dev/sdb type: USB vendor: SanDisk model: SD9SN8W512G size: 465.76 GiB
Partition:
ID-1: / size: 911.75 GiB used: 166.72 GiB (18.3%) fs: ext4 dev: /dev/sda3 ID-2: swap-1 size: 3.73 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sda2
Sensors:
System Temperatures: cpu: 43.0 C mobo: N/A gpu: nvidia temp: 50 C Fan Speeds (RPM): N/A
Info:
Processes: 302 Uptime: 10m Memory: 15.65 GiB used: 4.43 GiB (28.3%) Shell: zsh inxi: 3.0.36
On my system R seams to work correctly.
System:
Host: gsk Kernel: 5.3.10-134.current x86_64 bits: 64 Desktop: Budgie 10.5.1 Distro: Solus 4.0
Machine:
Type: Desktop Mobo: Gigabyte model: B450 AORUS M v: x.x serial: <filter> UEFI: American Megatrends v: F1 date: 06/26/2018
CPU:
Topology: Quad Core model: AMD Ryzen 5 2400G with Radeon Vega Graphics bits: 64 type: MT MCP L2 cache: 2048 KiB Speed: 2221 MHz min/max: 1600/3600 MHz Core speeds (MHz): 1: 1724 2: 1670 3: 2237 4: 2209 5: 2475 6: 2227 7: 2783 8: 2592
Graphics:
Device-1: AMD Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] driver: amdgpu v: kernel Display: x11 server: X.Org 1.20.5 driver: amdgpu resolution: 2560x1440~60Hz OpenGL: renderer: AMD RAVEN (DRM 3.33.0 5.3.10-134.current LLVM 9.0.0) v: 4.5 Mesa 19.2.5
Audio:
Device-1: AMD Raven/Raven2/Fenghuang HDMI/DP Audio driver: snd_hda_intel Device-2: AMD Family 17h HD Audio driver: snd_hda_intel Sound Server: ALSA v: k5.3.10-134.current
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 IF: enp3s0 state: down mac: <filter> Device-2: Intel Wireless 7260 driver: iwlwifi IF: wlp4s0 state: up mac: <filter>
Drives:
Local Storage: total: 704.24 GiB used: 167.24 GiB (23.7%) ID-1: /dev/sda vendor: LITE-ON IT model: LMT-256M6M mSATA 256GB size: 238.47 GiB ID-2: /dev/sdb vendor: Seagate model: ST500LM000-1EJ162 size: 465.76 GiB
Partition:
ID-1: / size: 17.72 GiB used: 15.73 GiB (88.8%) fs: ext4 dev: /dev/sdb2 ID-2: /home size: 210.72 GiB used: 151.51 GiB (71.9%) fs: ext4 dev: /dev/sdb3 ID-3: swap-1 size: 14.90 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdb4
Sensors:
System Temperatures: cpu: 44.6 C mobo: N/A gpu: amdgpu temp: 44 C Fan Speeds (RPM): N/A
Info:
Processes: 246 Uptime: 48m Memory: 11.71 GiB used: 1.83 GiB (15.6%) Shell: bash inxi: 3.0.36
I'm wondering if it's an intel vs amd issue above. Does anyone that run AMD have this problem with R or is it intel only?
Just tested on an AMD Ryzen 7 3700X, and it output the correct p-value for aov. System info to follow.
System:
Host: serebit-desktop Kernel: 5.4.1-137.current x86_64 bits: 64 Desktop: Budgie 10.5.1 Distro: Solus 4.0
Machine:
Type: Desktop System: Micro-Star product: MS-7B85 v: 1.0 serial: <filter> Mobo: Micro-Star model: B450 GAMING PRO CARBON AC (MS-7B85) v: 1.0 serial: <filter> UEFI: American Megatrends v: 1.B2 date: 10/28/2019
CPU:
Topology: 8-Core model: AMD Ryzen 7 3700X bits: 64 type: MT MCP L2 cache: 4096 KiB Speed: 2199 MHz min/max: 2200/3600 MHz Core speeds (MHz): 1: 2199 2: 2197 3: 2199 4: 2200 5: 2199 6: 2199 7: 2200 8: 2199 9: 2198 10: 2198 11: 2198 12: 2200 13: 2199 14: 2199 15: 2199 16: 2200
Graphics:
Device-1: AMD Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] driver: amdgpu v: kernel Display: x11 server: X.Org 1.20.5 driver: amdgpu,ati unloaded: fbdev,modesetting,vesa tty: N/A OpenGL: renderer: AMD Radeon RX 480 Graphics (POLARIS10 DRM 3.35.0 5.4.1-137.current LLVM 9.0.0) v: 4.5 Mesa 19.2.6
Audio:
Device-1: AMD Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] driver: snd_hda_intel Device-2: AMD Starship/Matisse HD Audio driver: snd_hda_intel Device-3: Blue Microphones Yeti Stereo Microphone type: USB driver: hid-generic,snd-usb-audio,usbhid Sound Server: ALSA v: k5.4.1-137.current
Network:
Device-1: Intel Wireless-AC 9260 driver: iwlwifi IF: wlo1 state: down mac: <filter> Device-2: Intel I211 Gigabit Network driver: igb IF: enp34s0 state: up speed: 1000 Mbps duplex: full mac: <filter> IF-ID-1: virbr0 state: down mac: <filter> IF-ID-2: virbr0-nic state: down mac: <filter>
Drives:
Local Storage: total: 5.23 TiB used: 2.32 TiB (44.4%) ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO 500GB size: 465.76 GiB ID-2: /dev/sda vendor: Samsung model: SSD 850 EVO 250GB size: 232.89 GiB ID-3: /dev/sdb vendor: Western Digital model: WD10EZEX-08WN4A0 size: 931.51 GiB ID-4: /dev/sdc vendor: Western Digital model: WD40EZRZ-00GXCB0 size: 3.64 TiB
Partition:
ID-1: / size: 453.31 GiB used: 163.82 GiB (36.1%) fs: ext4 dev: /dev/dm-1
Sensors:
System Temperatures: cpu: 50.6 C mobo: 38.0 C gpu: amdgpu temp: 41 C Fan Speeds (RPM): fan-1: 0 fan-2: 922 fan-3: 846 fan-4: 938 fan-5: 0 fan-6: 0 gpu: amdgpu fan: 882
Info:
Processes: 358 Uptime: 1h 40m Memory: 15.65 GiB used: 3.71 GiB (23.7%) Shell: bash inxi: 3.0.37
I think you guys were right and it is a hardware issue. I had an old surface pro 3 laying around so I threw solus budgie on it, brought it up to date, ran sudo usysconf run -f and rebooted a few times to make sure it was stable. Then I installed R and rebooted a few more times. Then I ran tools::testInstalledPackages(scope = "base") and the tests failed at the stats package as expected.
Here is the inxi -Fz output from the surface:
System:
Host: adurante-surface Kernel: 5.4.1-137.current x86_64 bits: 64 Desktop: Budgie 10.5.1 Distro: Solus 4.0
Machine:
Type: Laptop System: Microsoft product: Surface Pro 3 v: 1 serial: <filter> Mobo: Microsoft model: Surface Pro 3 v: 1 serial: <filter> UEFI: American Megatrends v: 3.11.2150 date: 04/26/2017
Battery:
ID-1: BAT0 charge: 30.6 Wh condition: 39.7/42.2 Wh (94%)
CPU:
Topology: Dual Core model: Intel Core i7-4650U bits: 64 type: MT MCP L2 cache: 4096 KiB Speed: 998 MHz min/max: 800/3300 MHz Core speeds (MHz): 1: 961 2: 938 3: 989 4: 962
Graphics:
Device-1: Intel Haswell-ULT Integrated Graphics driver: i915 v: kernel Display: x11 server: X.Org 1.20.5 driver: intel unloaded: fbdev,modesetting,vesa resolution: 2160x1440~60Hz OpenGL: renderer: Mesa DRI Intel Haswell Mobile v: 4.5 Mesa 19.2.6
Audio:
Device-1: Intel Haswell-ULT HD Audio driver: snd_hda_intel Device-2: Intel 8 Series HD Audio driver: snd_hda_intel Sound Server: ALSA v: k5.4.1-137.current
Network:
Device-1: Marvell 88W8897 [AVASTAR] 802.11ac Wireless driver: mwifiex_pcie IF: wlp1s0 state: up mac: <filter> Device-2: Marvell Bluetooth and Wireless LAN Composite Device type: USB driver: btusb
Drives:
Local Storage: total: 238.47 GiB used: 5.89 GiB (2.5%) ID-1: /dev/sda vendor: Samsung model: MZMTE256HMHP-000MV size: 238.47 GiB
Partition:
ID-1: / size: 229.59 GiB used: 5.89 GiB (2.6%) fs: ext4 dev: /dev/dm-1
Sensors:
System Temperatures: cpu: 59.0 C mobo: N/A Fan Speeds (RPM): N/A
Info:
Processes: 208 Uptime: 5m Memory: 7.75 GiB used: 1.03 GiB (13.3%) Shell: bash inxi: 3.0.37
So with this machine included:
CPUs of Broken Installs
Me (laptop 1): Intel i7-8700K - R install broken stats package
Me (surface): Intel i7-4650U - R install broken stats package
@kccit : Intel Core (Skylake IBRS) - R install broken stats package
CPUs of good installs:
Me (laptop 2): AMD A12-9700P - R installed no broken packages
@serebit : AMD Ryzen 7 3700X - R installed no broken packages
@gs : AMD Ryzen 5 2400G - R installed no broken packages
@joebonrichie since earlier you said you weren't able to replicate the incorrect results, I am willing to bet your on an AMD CPU as well?
@joebonrichie @DataDrake @adurante
FYI, I'm not sure if it's caused by Intel vs AMD issue. However, if that is indeed the issue, the problem maybe Intel optimizations solus did as I loaded up 2 VMs (ubuntu and void linux) - fresh installs. Both passed the R tools::testInstalledPackages(scope="base") and a simply lm() simulation with no problems.
Broken R here as well with:
System:
Host: tomaatti Kernel: 5.2.13-126.current x86_64 bits: 64 Desktop: Budgie 10.5.1 Distro: Solus 4.0
Machine:
Type: Laptop System: HP product: HP EliteBook 830 G5 v: N/A serial: <filter> Mobo: HP model: 83B3 v: KBC Version 04.53.00 serial: <filter> UEFI: HP v: Q78 Ver. 01.04.00 date: 09/12/2018
Battery:
ID-1: BAT0 charge: 46.9 Wh condition: 46.9/46.9 Wh (100%)
CPU:
Topology: Quad Core model: Intel Core i5-8350U bits: 64 type: MT MCP L2 cache: 6144 KiB Speed: 800 MHz min/max: 400/3600 MHz Core speeds (MHz): 1: 800 2: 800 3: 800 4: 800 5: 800 6: 800 7: 800 8: 800
Graphics:
Device-1: Intel UHD Graphics 620 driver: i915 v: kernel Display: x11 server: X.Org 1.20.5 driver: intel unloaded: fbdev,modesetting,vesa resolution: 1920x1080~120Hz OpenGL: renderer: Mesa DRI Intel UHD Graphics 620 (Kabylake GT2) v: 4.5 Mesa 19.2.1
Audio:
Device-1: Intel Sunrise Point-LP HD Audio driver: snd_hda_intel Sound Server: ALSA v: k5.2.13-126.current
Network:
Device-1: Intel Ethernet I219-LM driver: e1000e IF: enp0s31f6 state: down mac: <filter> Device-2: Intel Wireless 8265 / 8275 driver: iwlwifi IF: wlp1s0 state: up mac: <filter>
Drives:
Local Storage: total: 2.05 TiB used: 22.94 GiB (1.1%) ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB256HAHQ-000H1 size: 238.47 GiB ID-2: /dev/sda type: USB vendor: Seagate model: ST2000LM007-1R8174 size: 1.82 TiB
Partition:
ID-1: / size: 145.52 GiB used: 22.94 GiB (15.8%) fs: ext4 dev: /dev/nvme0n1p7 ID-2: swap-1 size: 3.73 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/nvme0n1p6
Sensors:
System Temperatures: cpu: 45.0 C mobo: 40.0 C Fan Speeds (RPM): N/A
Info:
Processes: 246 Uptime: 5h 11m Memory: 7.70 GiB used: 3.44 GiB (44.6%) Shell: bash inxi: 3.0.34
After this I updated the kernel 5.3.18-140.current and updated all packages but the issue remains.
I have quite a bit limited knowledge of Linux but nevertheless I've been thinking whether there could be issue with the OpenBLAS library (after a talk with a stats TA). La_library() function in R gives me "/usr/lib64/haswell/libopenblas_haswellp-r0.3.2.so", and I'm quite sure the Ryzen systems have something different, so maybe? But more curiously I would be interested to know what do the functioning VM setups have? Due to my limited knowledge, I haven't yet managed to do any testing with a different BLAS library in R (not familiar with compiling...yet) or do some similar linear calculation with Python and Numpy (I think Numpy uses the same library) and see how they compare to the results in R.
But this of course is just wildly hypothesising.
Actually, the issue seems to be caused by the libopenblas_haswellp-r0.3.2.so OpenBLAS library. I decided to remove a symbolic link pointing to that library (= /usr/lib64/haswell/libopenblas.so.0), and the R reverted to using /usr/lib64/libopenblas_core2p-r0.3.2.so. Now I get a correct result from my reference calculations.
Before:
> lm(Sepal.Length~Sepal.Width+Species, data=iris)
Call:
lm(formula = Sepal.Length ~ Sepal.Width + Species, data = iris)
Coefficients:
(Intercept) Sepal.Width Speciesversicolor Speciesvirginica
-1.1562 -0.3158 11.5719 11.6048After:
> lm(Sepal.Length~Sepal.Width+Species, data=iris)
Call:
lm(formula = Sepal.Length ~ Sepal.Width + Species, data = iris)
Coefficients:
(Intercept) Sepal.Width Speciesversicolor Speciesvirginica
2.2514 0.8036 1.4587 1.9468And the latter is the correct result I get on Windows and through semi-manual calculation.
Of course I have no idea why using libopenblas_haswellp-r0.3.2.so produces the incorrect results, but it seems to be the culprit on my system. Could someone verify this?
edit: put the wrong "Before results", now corrected.
@Voltti I am also bad at Linux but trying to learn :)
I can check on my machines however could you let me know how/where you changed the symlinks from one openblas library to the other? I have never done this in the past.
@adurante
What I did was:
> sudo unlink /usr/lib64/haswell/libopenblas.so.0
I have a feeling that this might cause issues with some programs, so, *I think*, you can redo the symlink with:
> sudo ln -s sudo unlink /usr/lib64/haswell/libopenblas.so.0 /usr/lib64/haswell/libopenblas_haswellp-r0.3.2.so
Yeah, this is my first time playing with these as well :)
After some googling, I don't think that would relink the other openblas library.
from this command ln -s source_file myfile I believe source_file should be /usr/lib64/haswell/libopenblas_haswellp-r0.3.2.so but myfile should be relating to wherever R is calling the openblas library (not that I know where this is). I could be completely wrong however.
How did you confirm that R defaulted to the other openblas library in your test after unlinking the /usr/lib64/haswell/libopenblas.so.0? This way I can make sure I confirm the same result on my end once i get to my other machine
@adurante
I think you are right about the relinking.
For checking what library R uses for BLAS I used the La_library() function in R console.
I can confirm that the command sudo unlink /usr/lib64/haswell/libopenblas.so. 0 fixes the issue on my machine.
I will note however before running this command the output from R's La_library() command was /usr/lib64/haswell/libopenblas_haswellp-r0.3.2.so and after it was /usr/lib64/libopenblas_core2p-r0.3.2.so
However I have no idea what this might break
That particular library is provided by the OpenBLAS package, so there could be some issues with other applications as well - I have numpy and Inkscape that rely on OpenBLAS libraries.
openblas has been updated to 0.3.7 since, can you confirm you are still getting the wrong results with the /usr/lib64/haswell/libopenblas_haswellp-r0.3.7.so library?
I should also clarify that currently the /usr/lib64/haswell subdirectory is only used by intel processors. So even if you have an AMD processor with the required instructions it won't load from the haswell subdirectory. I wasn't aware of this previously.
It seems that has fixed the issue on my system!
> La_library() [1] "/usr/lib64/haswell/libopenblas_haswellp-r0.3.7.so"
> install.packages("Devore7")
Installing package into ‘/home/adurante/R/x86_64-solus-linux-gnu-library/3.6’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/src/contrib/Devore7_0.7.6.tar.gz'
Content type 'application/x-gzip' length 493681 bytes (482 KB)
==================================================
downloaded 482 KB
* installing *source* package ‘Devore7’ ...
** package ‘Devore7’ successfully unpacked and MD5 sums checked
** using staged installation
** data
*** moving datasets to lazyload DB
** inst
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (Devore7)
The downloaded source packages are in
‘/tmp/RtmpWVRrLz/downloaded_packages’
> library(Devore7)
Loading required package: MASS
Loading required package: lattice
> motor_stack<-stack(ex10.37)
> motor_aov<-aov(values~ind, motor_stack)
> summary(motor_aov)
Df Sum Sq Mean Sq F value Pr(>F)
ind 4 30.86 7.714 8.444 0.000187 ***
Residuals 25 22.84 0.914
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1And all tests now pass!
> tools::testInstalledPackages(scope = "base") Testing examples for package ‘base’ Testing examples for package ‘tools’ Testing examples for package ‘utils’ Running vignettes for package ‘utils’ Testing examples for package ‘grDevices’ Testing examples for package ‘graphics’ Testing examples for package ‘stats’ Testing examples for package ‘datasets’ Testing examples for package ‘methods’ Testing examples for package ‘grid’ Running vignettes for package ‘grid’ Testing examples for package ‘splines’ Testing examples for package ‘stats4’ Testing examples for package ‘tcltk’ Testing examples for package ‘compiler’ Testing examples for package ‘parallel’ Running vignettes for package ‘parallel’
I updated OpenBlas package and I'm now using /usr/lib64/haswell/libopenblas_haswellp-r0.3.7.so in my R, and the reference calculations that I've been using(see my comment above) gives the correct result.
Thanks, I modified glibc to allow loading from /usr/lib64/haswell on AMD processors but I wasn't seeing the incorrect calculations so I wanted to make sure. Thanks for your time spent debugging the issue. I'll mark as closed.
Resolved with R2228:0a2db4c6ca45.
Just to add: my test was done on Intel. I think those drivers are optimized for Intel processors.